Skip to content

0.1.3 Orchestration: cron and airflow #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: 3-012-viewing-data-through-duck-db-from-iceberg
Choose a base branch
from

Conversation

brmhastra
Copy link
Collaborator

defined cron functions

@tusharchou tusharchou added enhancement New feature or request good first issue Good for newcomers good first review Good for newcomers labels Oct 23, 2024
@tusharchou tusharchou added this to the 0.1.3 Orchestration milestone Oct 23, 2024
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start. I suggest you move this to a package eg: .../local_data_platform/cron/{your_python_script}
Let's say an end user wants to use any cron functionality then they would simply do
"""
from local_data_platform.cron import cron_flow
"""

Copy link
Collaborator

@redpheonixx redpheonixx Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Can we add logger module to add logs for eg. in exception part @brmhastra @mrutunjay-kinagi @tusharchou This way it will be easy to debug in case pipeline breaks.
above point is for all python scripts written till now and is not only subjected to this PR only.
we can add logger module in later stages as well.

@tusharchou tusharchou changed the title adding cron placeholder file 0.1.3 Orchestration: cron and airflow Oct 29, 2024
@tusharchou tusharchou marked this pull request as draft October 29, 2024 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers good first review Good for newcomers
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants